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SUMMAKY 

Fifty lino pllotu (cuptalna, firwt offlcnrn, ami fllylit laiginccra) Iroiii 
8 different airllnen were admin late red a atruetured qm i luiuuil re relating 
to future warning ayatera dealgn and aolutlonw to eurrenl warning ayatem prub*- 
letna* This waa followed by a aemantlc! differential to obtain a laettjr 
analysia of 18 different cockpit wurulng algnala on HcaUm nudi aa Informative/ 
distracting^ annoylng/aoothing. Half the plJotw received a dcanonatratlun el 
the experimental text and voice ayntheaiaer warning ayntema before annwerJug 
the questionnaire and the sewuntic differential. A control group annwi;red 
the questionnaire and the semantic differential first, thus providing a check 
for the stability of pilot preferences with and without actual exposure to 
experimental systems. It was hypothesized that preferences for w.irning method 
and cancellation method would vary as a function of warning urgency or pri“ 
orlty and as a function of expected false-alarm rate. It was also thought 
that age and position flown might influence pilot preferenci-s . There were no 
significant differences between the two groups for overall preferemms for 
text and voice warnings compared to other warning methods, suggesting a high 
degree of stability and reliability of pilot preferences for warning methods. 
Warning urgency and expected false-alarm rate did produce slgni£lc.ant differ*- 
ences in pilot preferences for some, but not all, warning methods. Warning 
urgency also produced significant differences in preferred canceliatlon 
methods for some warning methods. Generally, the preference data obtained 
revealed much consistency and strong agreement among line pilots concerning 
advance cockpit warning system design. 


INTRODUCTION 


There seems to be substfu. .ial agreement among members of the aviation 
community that current cockpit warning systems for comnuncial jet transports 
suffer from a wide range of human factors design problems, In a recent study 
by the Boeing Company (ref. 1), funded by the Ftaier.'ii Avi;itlon Admlni jitriit I on , 
Vletengruber documented the warning systems In aircraft now curnmt in tlic 
fleet. He found insufficient standardization of warning slg.nals hetween 


i^iThls research was supported by NASA Ames Research Centi’r Grant 
NGL-05-046-002, San Jose State University Foundation Account 02-0l-W,IA. 

^'he term "line pilots" refers to pilots who rcgulai Iv l ly immiioirial 
transport aircraft. It does not Include airline chock pilots, cliiel pilots, 
or instructor pilots. 
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aircroft typua and ovon for the anroc aircraft typo botv'ocm airllnea, Ho alao 
noted a trend toward IncreoalnB numbora of different warning algnala in both the 
visual and auditory nvodoa. George Cooper (poraonol communication), mider con- 
tract to the National Aeronautica and Space Adminiatration, conducted structured 
interviews with foreign and domestic aircraft and avionico manufacturers in 
order to document current philosophies and to identify specific guidelines which 
might assist in improving warning system design. He found general agreement 
that cockpit warning systems are inadequate and may be adding to cockpit work- 
load at times when this is already heavy duo to additional demands on crow 
attention. While general agreement was found for many guidelines, the Cooper 
study also revealed some major points of disagreement among the parties inter- 
viewed regarding preferred methods of alerting, that is, tones, bells, voice, 
alphanumerics, labeled lights, and tactile warnings such as the stick shaker. 
However, a need for improved standards or guidelines was recognized. 

There are a number of proposals that define general approaches to cockpit 
warnings and assign particular alerting methods to particular hazardous condi- 
tions. Aeronautical Radio, Inc, (ARINC) has a project paper (ref. 2) that out- 
lines specifications for airborne audible warning generators. By assigning 
specific aural alerts to specific hazardous conditions, they have attempted to 
standardize the “meanings" that pilots would have to learn to associate with 

each of the different types of sounds. They also recommend a visual annunciator 

that would remain on until an existing fault is corrected. They provide for the 

possibility of voice warnings in place of or in addition to the nonspeech aural 

warnings. The Society of Automotive Engineers (SAE-7) Steering Committee on 
cockpit design is currently working on design standards for future warning 
systems. British Airways, in a paper for the International Air Transport 
Association (ref. 3), presented strong arguments against the use of nonverbal 
aural alerts, pointing out that such alerts are limited in the amount of 
information they can transmit and often are startling and distracting. They 
suggest audio alerts, preferably voice, for high priority, quick-action dangers 
and visual warnings, color coded for priority, for all priorities of warnings. 
They recommend that high priority voice warnings be non cancellable while pro- 
viding for a cancel button for lesser priority voice warnings. Clearly, there 
is no industry-wide consensus regarding the types of alerting signals or the 
system logic that should be used for cockpit' warning systems. 

One type of data frequently overlooked is objective measurements of user 
preferences. All too often, experimental ^vs terns are designed and tested in 
the simulator first, with pilot debriefings afterwards. Perhaps this approach 
is popular beuause of a belief that there is little agreement among pilots con- 
cerning new cockpit displays and therefore little to be gained by asking them 
what they want in advance. This deprives an investigation of the vast resource 
of flying experience of the pilots who are destined to use and depend on the 
new system until major commitments have been made to particular design 

elements or types of systems. This study had the dual purpose of sampling line 
pilot preferences for cockpit warning system design and also providing data 
that would be useful in guiding subsequent flight simulation research aimed at 
the determination of design principles for warning systems for air transport 
aircraft. 
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HYPOTHESES 

some eanoral 1 -ypothesca .egordljg 

were Cro£. 4) and on voUe »orning systom design 

?nU^g^S«y ^n syntHeslsed speech displays. 

During discussions with pilots on f ' 

(GPUS). It was observed that ‘ iupjttate thalr opinion. This 

toward voice warnings and used , prticXes appearing In publications 

Observation was ““P^vllrtlnf pllotrciras! "fis! an" 1976 (refs. 6 

written for and read ^ \gj^ed if voice warnings with extremely low 

and 7). However, ^ ^ many responded that voices could be very 

false-alarm rates would be acceptabl , , ^„_„nd on them to be accurate, 

useful for high priority warnings if one f per se, but 

SSruid 5:prb^ on“thn^^^^^^^^^^ ^ol theVle« signaled by 

the warning and on the expected fa se a a 

Another point that pilots -phaslsed^ Slt^^Suirdrrlnrdaoislon! 
difficulty created by loud aomds an vo ^ frequently given 

making and intracrew communication. The , attention to the decision- 

Z an%xample of a signal that prevented or disrupted ^ttention^t^^^^ 

making process “asked crew checklist comments about cancellation 

events. On the other hand, pi^-ots gave ^P, . other visual warnings 

of visual warnings. Some wanted to visual displays, 

as soon as they occurred to prevent f conditions 

Others wanted visual signals to ® . hypothesis that preferences for 

remained. From these ^otf Lr^i^ 

warning cancellation would depend f„ addition, it was hypothesized 

method used (i.e., auditory, , visual signals would satisfy the 

that a limited priority assignment scheme for visual signaxe 

majority of pilots. 

1 t-^A t-Viaf ane and position flown might have an effect on 

It was also expected that age an p no specific predictions 

^c“"Xnsrprsir^rc?r:: .v-Lice. fhc„. ih. 

hypotheses were that 
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l. Pilot preferoncos for warning method will depend on the 
problem algnaled and on the expected falao-alnrm rate. 


urgency of the 


2. Preferencea for warning cancellation will depend on warning 
the warning method, ^ 


urgency and 


3, A limited priority assignment scheme for visual 
most pilots. 


signals would satisfy 


4, Age of pilots and crew position flown might produce differences In 
preiereuc 08 • 


4 


® pilot preference survey Is proposed, resistance Is encoun- 
tered because of supposed characteristics of pilot preferences. In fact, these 

this, subjects were offered reasonable alternatives to rate or rank or were 
off«ra choices among alternatives. In most cases, they were also 

or responses where they could write in their own opinions 

InadeoLtP system, if they thought the ones offered were totally 

!* If this alternative was used by a significant proportion of 

be evidence of an unmanageable diversity of opinion among 
pilots. Inadequate test items, or both. f among 

Further, it is often suggested that pilot opinions are too changeable 
because they consider whatever new system they saw last to be best. \his prob- 
maklne^fl ^ ^ Splitting the sample group into two subgroups and then by 

^ten?l!l ‘'he subgroups of the usefulness^and 

irSe^tLf m f systems, CRT or voice, which would then be represented 

suberLfr n significant differences between the responses of the 

p!!efereSLrob1SLr changeability of the pilot 


Measuring Instrument 

i-oof instrument was a 32-page booklet to be filled out by each 

5rfrefJe«f‘n f 2 pages of biographical information and 30 pages 

o free response, rating scale, preference grid, and ranking Items. There was 
a second, optional test, the semantic differential. In a separate booklet. It 
consisted of Judgments of 16 concepts on 17 polar opposite scales. It was only 
administered to subjects who finished the first booklet within the 3 hr allowed 
compleL!°“*’^^ session. If administered, it took approximately 15 min to 
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SubjpotH 

Sul»Jpcl(i wt'i’o Unc captalnH (13), flmt ofl JcurB (20), or flight engineers 
(1/*) currently flying transport aircraft or recently furloughed (3). The 
oldest was 60, t1u> youngest 27, with a mean age of 41.3 years. Total time 
averaged 9300 hr, and 8 airlines were represented. (See table 1.) This sample 
was not randomly chosen, and so may not represent a true cross section. 


Procedure 

Subjects were obtained in sets of 1 to 9 persons. They were drawn from a 
pool of airline pilots based in the San Francisco Bay Area who had expressed 
interest in participating in research at Ames Research Center. They were paid 
for their participation. Each set was randomly assigned to treatments, except 
that the last set was picked to exactly complete the group sample size of 25. 

V^jnoi'UtAdtion-^-Out gkoup — Sets of pilots who were assigned to the 
"demonstration-first” group were offered coffee and a short, purposely vague 
ihtroduction^ to the purpose of the study, and assurance that their name and 
airline could not be connected to their individual responses on the test items. 
They then participated in a 20-mln experiment that exposed them to synthesized 
speech warning messages. Following that, they were shown a video tape of 
various possible CRT display warnings, and then color slides of this type of 
CRT system and another type of alphanumeric warning system in several simulators, 
aircraft, and artists* conceptions. Finally, they were given the two test 
booklets and allowed to work on them at their own pace, for a maximum of 3 hr. 
Most completed them in less time. 

Quests onn(UAe.-jj-cUt gAoup — The **questionnaire-first" pilots were given 
the short introduction, coffee, anonymity assurances, and told they would have 
2 1/2 hr to complete the test booklets, after which they were given the syn- 
thesized speech experiment and the video and slide demonstrations. 

The experimenters were always present during administration of the test 
booklets to answer questions. Discussions were not allowed to become estab- 
lished. If opinions were offered spontaneously, subjects were politely 
encouraged to write them in the appropriate spaces on the booklets. Motivation 
was good, and subjects willingly worked on the booklets without complaint. 
Spontaneous comments offered at the end of the session were encouraging. All 
subjects were offered an opportunity to fill out a name and address sheet to 
receive a copy of the report on the study. More than two-thirds of them chose 
to do so; 41 of the 50 also chose to do the optional semantic differential. 

Tliese are rough Indications of good motivation and interest. 


^The Introduction was left vague to prevent biasing subjects. If any 
particular system was even mentioned In a positive or negative statement, it 
could have influenced their responses. 
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RESULTS AND DISCUSSION 

In the following, whenever a reauit Is pren(?riteii oj. a eoncluiiiou ollcrcil, 
aHHume that the "demonatratlon-f irst" and the "questlonnai re-f 1 rat" gtnu|)H 
were not significantly different, and the data were eomlrlned after the Hlgnll- 
icancc tost had been run. The few instances when there wan a nLgnlf leant dif- 
ference are indicated. The tcnn "pilots" Is used to describe any of tlie sub- 
jects, whether they were captains, first officers, or flight eiigtne(>rs. 

Throughout this paper, there are repeated references to priority 
(« urgency) levels, 1 through 5. Tliese were adapted from a priority assignment 
scheme developed by the Boeing Company and were presented to subjects in the 
form shown in table II. Warning methods are repeatedly referred to, and these 
were initially presented to the subjects as shown in table Hi. Subsequently, 
they were referred to in an abbreviated form. 


Warning Method Preferences 

The preference grid shown in table IV was repeated four times, once for 
each of four false-alarra rates? 50 false: 1 real alarm, 1 false:50 real alarms, 

1 false: 1000 real alarms, and 1 false: 1,000,000 real alarms. Pilots were asked, 
for each of these rates, to place an X under the system or systems they would 
want for warnings of that urgency, given that the system false-alarm rate could 
be no better than stated. Results are shown in figure 1. There is much 
information in this figure, but it will reward close study. First consider 
one cell concerned with voice warnings for priority 1 problems (fig. 2). Note 
that with the high false-alarm rate, 50:1, few respondents will accept the 
voice warning, but as the false-alarm rate improves, more and more pilots are 
willing to accept this method of warning for priority 1 situations. Returning 
to figure 1, the larger histogram, note, in the row for voice warnings, that 
for the lower priority situations, the number of pilots desiring voice v^arnings 
declines, regardless of the false-alarm rate. In short, pilots do not want 
voice for "information only" or low priority warnings, and they do not want 
them if the false-alarm rate is high. But they are willing to accept voice 
warnings for very important, high priority warnings if the false-alarm rate is 
low. 


Now consider the row "text message." Note that It Is generally acceptable 
no matter what the false-alarm rate is, as shown by the evenness of the histo- 
grams within cells. But also note the slightly greater concentration of 
responses within the cell for urgency level 3. This v/ould tend to indicate 
that text messages arc seen as more valuable for moderate priority iti’ms. 

Two further observations can be made from this figure. For the auditory 
warnings (top three rows), responses are concentrated in the higlier priority 
columns, indicating that pilots want sounds only lov iuiportatit prohK'ms. 

A test for the effect of false-alarm rate Is also <'oiu! istont ( i/- = 22.6, 
df =12, p < 0.05). 
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Now consider visual displays. A n-lallvc Inscric. J I I v 1 1 y to 1 . 1 1 r.i •• ,i I ai m 
rates and urgency level is sliown. These responses snggi'sL tliai visual w.nnisp.s 
con he more easily Ignored i£ they are a false alarm, and that Hiey < as I" 
tailored to suit the urgency of the situation. '//■ tests weia perlonn.-.l mi ilv 
data for each false-alarm rate to test tlie relation ol warnlnp, meihoil lo 
urgency level. For all false-alarni rates, sign 1 flcain'e levels, of p O.OS m 
bettor were obtained. Inspection of the tabled data allowed tli.it the .luditoiy 
methods, particularly "other sound" and "voice," were ))relerred lor 1 he imu r 
urgent warnings provided the false-alarm rate was low. Visual method!;, )>ar 
tlcularly "labeled light" were preferred for moderate and less urgent w.-miings 
for all false-alarm rates and were also preferred for level 1 and 2 urgency, 
"action now" warnings when the false-alarm rate was high. 

When differences between the responses of the "questionnaire-first" and 
"demonstration-first" groups were tested on this item, none of the expecti.-d 
ones were found, despite this being one of the most likely sections ol tlie 
questionnaire to show such differences. 

Warning Cancellation 

Part of matching a warning to a situation is providing a way to cancel the 
warning when it is no longer wanted. Responses were collected by means of a 
preference grid with cancellation options and warning methods, given dillerent 
priorities. The results are shown in figure 3. For clarity, one cell is shown 
in figure 4, voice warnings for priorities 1 and 2. (For brevity on the ques- 
tionnaire, priorities 1 and 2 were combined as were priorities 3 and 4.) For 
these, "cancel button" is the method most preferred, and this is a general 
finding. 


Note the very small number of "noncancellable" responses, vdn'ch Indicate 
that the respondents do want to be able to cancel a voice warning. This is 
true for nearly all auditory war rirrgs . Also , few sub j ects checked "don ’ t u.se 
for this warning method, for this priority, indicating that they do find voic 
warnings acceptable. Figure 3 also shows that "noncancellable" is very fre- 
quently checked for lights, text messages, and flags; therefore they should 
stay on until the problem is solved. When x tests were applied to test for 
a statistically significant relation between urgency and cancellation pref<'i- 
ences, they were significantly related (p < 0.001) for auditory warning inetln 
but not for visual methods (p > 0.20). 

Finally, figure 3 shows that as priority decreases, there are m.niv more 
"don’t use" responses for audio displ;iys, the trend being reversed lot 
unlabeled lights and flags. 


System Logic 

Another question ;it is.sue in the design ol warning, syiiti'in!'. Is tin 
ment of priorities for the warnings and filtering or Inhibiting lliein. 
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jirohlnn v;/ih mmiiphiumI In llin rnnt<'>?t nj n f»hi}’lr (lianncl warning; that 

(M)iil(l not profUM)!. liin.IMpIo n Imti 1 1 anorun; wafulnpn* IMlc/tn w^To nnk(>d tu ratc^ 
tho Inllnv/liip, nyhli'iiin: 

A. An nnl>nnr<l rompnlri’ (I(m’I(Ic}; pMorlly, j)!cjaMili; tln» inout urHont warning 
vmt I 1 Uio rnmiltJon In roiiiovodj Ihon }n ofn iil n l)io noxl inont urfpaiti otc* 

U. A pr luiaj y ilJuplay j)roHrnln I ho monl nrfomi v/arnlnp.n; /i Htibn ld lnry dJn- 
play pr(‘n(‘nt.i] /my (‘thorn v/hlclj lanmr f; liniil I anonmi 1 y. 

h, Prlorlllcm arc* not ana I j'iu*d - v/ai n Injp] /iro pronontrd on tho primary 
cliapjay an tln*y ooour, 

!)• A wanriug In i)roHonLud JOr 5 soooiuls on tlu* primary display then 
roplacunl with am»ther warning, until all warnings in llu* stac’k are exhausted; 
then the entire thing repenats until /ill ('oiui i 1 ions retpiiring warnings are 
removed* All warnings are also d Is]‘l /lyod on /i suhsidl/iry dlspl/iy* 

E* An (inboard romputer /m/ily^.c‘s the p/ittoin of w/irnings, then presents the 
crew witli the hc*sl (‘oursc* of nvtAnn in eoimmind format* 

Two primary displ/iy systems wi.*re t(‘ he considered — nn "alphanumeric display 
block of 3/4- ln**~h:igh letters"^ or ;i "synthesized voice display in earphones and 
speakers*" Results arc* presented in figure 5* In all cases, the visual warning 

was slightly more desirable** System R is the f/ivorlte* The other systems 

hav(e mean ratings of 3, "no preference" or worse, so the ])ilots seem to indicate 
that they do not li,ke any of these, other systems very much. Clearly, more 
thought must be given to priority /issignment schemes; perhaps systcam B could be 
used as a starting point. 


T(*xt Displays V?ith /nul Without Alerting Tone 

The next item eoncerned the* use of a fl/ishing versus a nonflashing display 
and the use of an auditory alerting tone* A grid was presented showing differ- 
ent systems and urgency levels (table V)* Subjects went through the grid twice, 

first making an X for /my systems they would want for warnings of a given 

urgency level, /nul the second time m.aking an A for any systems they would 
want for a given urgency level It a single /mdjo /ilerting tone were presented 
at the same time* The results for /i w/irning on the bottom line of a CRT are 
shown in figui'e 6* Tin* "m^t flashing" viM^sion is most dc*slred for low-urgcncy 
warnings, wliilt* the audio tone is m‘t c<insidi*red p/ir t i cularly helpful for any 
priority* When the same warning Is fUishinp.* htnv't'ver, the preferences move to 
priority 2 with /uidio ;ilf*rt hig /nul to priority 3 withotit. 

Next c(.>t\sld(*r tigure /, "w/irning en whole CUT s«;reen." Wien this Is not 
fl/isliing, tin* preference's eonler .ireuml pi iorilv 2; but wln.'u it is flashing, 
the pitelerences inovi* in prifullv 1, llu* m(»st tu s,o»it Might <s/itety items. For 

tin ‘ M t • » lilt.' iiutl 1 1 1 .'1 1 1* V I i 11 ji', t ' n u ‘ 1 1 1 ;-i tt 1 u' ' I 'ittt-' ; ■ iiv ' i < ' t ^ t * s i i .'il' 1 1 * ♦ 

lA 1 1 inatetiol enelosed in fpiot/ition iii:n ks i;; a d i riu t (juol/iticm from the 
tp t. i (itn ) a i re » 

sy/i 
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Figuru 8 HhowH Uu». reniiltM Ini a iJm* oj twnlvn iiu /ilplia- 

numoricf?/’ AgaiUi wluni not: :Il If; vInv/oU m\ a Uovioo ho jo tssmil. i n); 

normal information or Uoih vital warnlugH, prloi Jty 1 or hoJnv;; in iliia t/nw, 
no audio alort In wantru]. V/hon riaalihph llioja* a.1 iJhmiuuuo irji «*an Im u;,od 
the hlght'Qt priority UtauH* In tliJn o/iho, llio audio tt»nc* Krt onuo; /ih I o an 

an alerting device* It f;ht>u]d he nolod tlial nomo oJ (In* doninina rat ( ofi giv<‘u to 
the subjects connlnted of J i I nntral- 1 oun of exact ly tlieru' t)u i‘o Myjitemj.* Thev 
were shown by video tape on a 12*7-’('m (')“:Iu*) by 14*0-cm (S*')-ln*) (JKJ' moniior 
and by color sllden ol tlu‘ 12”-a I plumumei J c bJiJck n,^;ed In on lli<* 

at Ames Resuarcli Cimter* Surpr Joingly , no HignJJicanI d 1 f fi»ronci*n Ju rcj;pom;eH 
were noted between the* "demount rat lon-^ri rat’* and "(juefa lonnai ro’^l i i nt" groupa 
for these items* 


Vojcf Warnings 

Another group of questions concerned voice warning systems* A technoJugi- 
cally possible future system was described Jn which each of the present aural 
warnings would be replaced by a voice warning with a different voice message 
for each different malfunction* The description is reproduced below. 

It would be possible to replace all of the current aural warnings with 
voice warnings* Such a system would be able to have different voice 
warnings for unsafe conditions which are now signaled by the same 
aural warning, e*g. TAKI‘M)FF warning and CABIN PRESSURE warning* Such 
a system would also include a volume control to adjust to different 
listening conditions. The warnings would be presented to your headset 
at the same volume as your own adjusted volume level for ATC coimnunica-* 
tions* As a back--up, the warnings would also be presente<l over a 
speaker with their volume automatically adjusted to be just suffi- 
ciently above the volume of the ambient cockpit noise so you would 
hear them clearly — as the noise level changed, the volume ol the 
warnings would be automatically adjusted up or down* A visual status 
display would also be included to display all unsafe conditions as 
long as they continued to exist* You would have the option of leaving 
the visual display on continuously or turning it on only when you 
wanted to look at it* There would be a cancel button for voice 
warnings . 

In response to the question, "Would you want such a system in your cockpit?", 
far more pilots said they would want the proposed voice warning system. Eighty- 
two percent of the 50 pilots gave a "yes" or a "qualilled yes" response while 
only 18 percent responded "no." This difference was highly slgnil leant as 
tested by the 50-Percent Probability test (x - 9, n - 50, p * 0,002). 

The pilot responses to the proposed voice warning system were potent tally 
among the most susceptible to possible influtnice from the demonst rat ion el 
experimental synthesixed speecli and text dii;pl.; s. In I igure 9, iln* ie;.pennes 
of the questionnaire-first group are shown on t ■ left, and tiiose ol the 
demonstration-first group are shown on the light The projuu i ions ol ’’ve: *' 
(including "qualified yes") to "no" rc'sponses wen iieai iy idi'iiti'ai ler tin two 


Thp only (H i It'rnuM* b('twi*rn l:liu two |.>roujJH 

proiKu l Inn nJ “quallfhui yrn“ roHpoania^ p1v(‘n. A nm/iUor jH’rrtMil /qv* of 
li'iipnunrM fni ( lu* doiiionii (rn 1 1 on -I h nt f»>roup wort^ ‘’qunl if Jr»d y(»n’’ roupniniinw 
Tin* pi loin (i) tin’ qunnt lonna I nil. p,roup» how^^vofi worn rt‘HpoinUnf\ on thn 

|hir»ln (»1 iin prior <»Hp(»rlonn' with tin* (Wip/ib j 1 1 1 I on ol ronio vuloo wnrnlnp; 

nvntonin, In thin oontnoU > It would ho roninmaliJo to find a h:lp,hor doppnn* of 
uin orlahHy oonooi nlnp t in* ohnr/iot (‘v tnf :l o.n <»t a pr(q)onod voloo warnJnp, nyfitom. 
Tliln, In turn, uould l»av/o o/iunod inor»* of thofn* plhftn to p,lvt* a **qualiliod yofi” 
i' 0 Hp<mf}o <’omparod t<» tho dojaonntrat Jnir d trnt ppamp. Ilov/ovor, thin apparent 
diliro'oaro holv/(n*n \.\u\ two p,rou]>n wan not nJpjrUitwml an nhown by FJnhor’ri tent 
(a H, h ’ 't, (* K!, (1 IH, p :• O.OS). 

Tin* next (|Ui‘nlIon wan doniguod to dotoniilnii whioli fuaturon or oumpomailB 
of 1 h(* )uop(tnod voliu* warning nyntom wt*ro ronponnihlo tor produoing *’y<jB'‘ or 
‘hiuallilod yen’* r(»yponnc!B to the nyntuaiu Figurt» 10 shown the percentage of 
‘'ownont i.a 1 ’’ jiuiguioutn rec’eivod by eacdi vn)icc‘ warnirig nystcnn component. These 
rtanponnen were given by the 41 pilutn who had responded affirmatively to the 
piruposed Hysteuu Kach pilot placed a (‘heck beside each component he thought 
was easeutiaJ to make the voice warning system ac!ccp table. The differences in 
numbers of yesaential‘* Judgments for the different components were highly sig- 
nificant: (x*^ - 54, df 6, p < 0.005). Clearly, the two most essential compo- 
nents are tlu» voice cancel button (65 percent) and the visual status display 
(63 percent) . We interpret this to mean that pilots want voice warnings only 
if they can cancel them and only if they have a visual status display that will 
continue to make the warning information available* Each of the remaining com- 
poueiits received some ’hissential** Judgments, but in each case, from less than 
50 percent of the pilots. These other components should certainly be regarded 
as desirable. In contrast, the voice cancel button and the visual status dis- 
play have to be included in any voice warning system* 

Figure 11 also supports the finding that pilots want to be able to cancel 
voice warnings. In this question, they were asked to choose among several 
types of voice warning repetition; 77 percent of the pilots wanted the warnings 
to repeat until they pressed a cancel button or the problem was corrected, 
whichever happened first. This compares to c^nly 19 percent who wanted the 
warnings repeated a fixed number of times — ’ once, twice, r three times — and 
a mere 4 percent who thought voice warnings should be noncancellable . These 
differences were highly significant by tests (x^ 44.8, df « 2, p < 0.001). 

The pilots as a group expressed no strong preference on the question con- 
cerning the effect of warning urgency on the type of voice warning repetition. 
Figure 12 shows that 39 percent of the pilots thought the type of repetition 
should depend on urgency level, while 61 percent thought urgency should not 
aff(*ct the repi^tition of voice warnings. This difference was not significant 
as l:<*sted by the 50--Percent Pia^bability test (x - 18, n 46, p > 0.10)* 

(Four pilnts did not respond to thiii (juestion.) Those pilots who felt voice 
warning repetition should depend on warning urgency wanted more repetitions 
and/ur more stringent cancellation conditions for high priority warnings than 
for Lower priority warnings. 


Tlic pi' iiliui piU'i! Uif-'Ir pri'lcTcncPH for tliP ufx'f-; of volcp warn Inf ',h 
( fif>, J I) , I'lnif, (In' nnmlH'r of p;l lotn who rtinpondotl /if flriivtf Ivoly io one or 
rooro of flio pi opoHoil unoH for volix* - /lU'rtl’ii', , iipocJfJc prohlom, limm'ill /ito 
.u,’(;loii, or "othor" ■ - w/m ooiiip/trod to tho nutnbor of pJlotn who rcHpondod "don't 
ufic" woh.'o. Thill t’/ivo II r/itJo of jiilotii roupomlliif, of f Irmativoly to only 
3 pi lotn ronpondlnf, "don't niio," Thlii dlficrcnco wan hJf.hJy nlp,ntf I cant an 
toiitod l.y tho ')0' I'orcont I'rohalrl 1 1 ty toat (x ® 3, n 46, p •' 0.002),'** Note 
thnt tho h-’poreont "don't uho" ronponnon comparu roughly witli tho ronpoiiHon to 
tho proponod voloo warning nyMtoiii whuro 18 poroont of tlio 50 pllota ronpondod 
"no." Thin o/in Ito (;d<on an an IntornaJ croanchock of the earlier finding that 
pilota gunerally are In favor oi the cudccpf of voice warningti. 

A threc'-way cotiipariBon of pilot affirmative reaponaea to the three proposed 
tuiu.tions of voices -• alerting, tell specific problem, and tell immediate 
actions — also resulted in significant differences in preferences (x^ “ 6,75, 
df ■=‘2, p 0.05). Mostly, the pilots wanted voice warnings to tell them the 
specific problem (78 percent). In addition, 64 percent wanted an alerting word 
such as "warning." Only 36 percent wanted to be told immediate action items. 
Under "other" uses (12 percent), suggestions were made by a few pilots that 
checklist items or immediate action items should be available on demand by 
voice or CRT display. These responses imply that a voice warning format con- 
sisting of an alerting word followed by a statement of the specific problem 
would be acceptable to most pilots. 


Age and Position Flown 

Neither age nor position flown resulted in significant differences for 
acceptability of the proposed voice warning system nor in the pilot ratings of 
the proposed text warning systems. Fisher's test for differences between 
younger (21 to 40 yr) and older (41 to 60 yr) pilots for the number of "yes" 
and "no" responses to the voice warning system yielded a=5, b=3, c<=19, 

d « 23, and p > 0,05. Similarly, a test for position flown (captain, first 
officer, or flight engineer) by number of "yes" and "no" responses yielded 
“ 1.09, df ■= 2, p > 0.10. 

The sums of individual pilot ratings for the five proposed visual text 
systems were also compared for the same younger and older pilot groups using 
Wilcoxon's sum of ranks test (n^ = 16, nj^ ® 16, ng “ 24, R = 305, z «= 0.63, 
p > 0.10). And, finally, a 3 >« 2 comparison of position flown by low (7 to 13) 
versus high (14 to 19) sums of ratings for the proposed visual text systems 
resulted In no significant effect for position flown (x^ ® 4.45, df = 2, 

p > 0.10). 


*^Kuur of the 50 pilots did not respond to this item. Assuming they had 
checked "don't use" the ratio of 43:7 would still have been significant at 
the 0.002 level. 


627 


Soranntii’ Dlfforcntlnl 


The (mnlynlfi of pilot ronponnoo to the nomnntlf^ hll Ifrciiil.il olim i cvc.dt'd 
Btrong ngroomont nmong pllotn rognrding tho Itutturcn of (In- ill i fcrtiii w;tnilii)> 
concoptfl. For thono rondorn unfnmiUnr with thin Innt rnnicn) , the <■ 

dlfforcntlni In n tcchnlquo plononrod by Ongood In Iffi/ (fid . H) . It In 
gonornlly unoful for finding related eonceptn In a dlvenie col le»( Ion. in nne, 
each concept or Itom to bo Judged Ifl placed at the (op of a page wlileh Imn a 
number of polar oppoalto acnloo. Tho concoptn uned here w(.'re fill wfiniing, 
related Itoraa (table VI). Tlio acaloB are Hhown In table VII. I'oibleein were 
given a 19-page booklet, 1 page of Inutrucllenn and IH pagen, ofieli wllh one 
concept. They placed an X on eucli ‘/-point ncale, e lonej to une ol I he pol.n 
opposite adjectives or tho other, depending on the one they lell the eoiecpt 
was most closely related to. If It was unrelated, or related to hoih /id,|eei .1 ven 
by the same amount, they placed an X In the middle space on tin- scfile. 

Data from the semantic differential arc usually anaJys'.ed several w/iyn. 

The analyses presented here Involve mean responses ol ail pilots on I'ach scale 
for each concept. Two-way comparisons between pairs of warning concepts fire 
shown in figures lA(a) through (g). Figure 14(g), for oxainple, shows tluit 
when "VASI lights" is compared to "Whoop, whoop, pull up, pull up," tlie. lights 
are less startling, more informative, far more beautiful, more valuable, more 
passive, far more quiet, and far more soothing. Thu useful conclusion, then, 
is that to startle and annoy one would use "whoop, whoop, pull up." To present 
Infoirmatlve unobtrusively, one would use VASI lights. 

Further use of the semantic differential for evaluation of o.xperimental 
cockpit warning systems seems warranted. Factor analysis teohnlqu,*s fire 
expected to extract groups of warnings that have similar values on the 17 polar 
opposite scales and to determine how factors such as evaluation, utility, and 
intensity characterize the different types of warnings. The aim is to standard- 
ize a set of semantic differential scales which could be used to characterize a 
new warning system in relation to existing systems merely by having pilots fly 
a simulation of the new system and then fill out a semantic differential 
booklet. 


CONCLUDING REMARKS 

Several conclusions can be drawn from the data analyzed and presented. It 
was shown that preferred warning methods depend on urgency or priority of the 
warned condition, and that false-alarm rate has a major Impact on the preferred 
presentation mode. For example, these data would indicate that, ll fi system 
has an Inherent high false-alarm rate, a visual warning method such ;is l ibeli-d 
lights or a CRT is preferable to any audio system. If fi low faUio-al.irm r.ite 
can be achieved, the audio systems, particularly voice, are prefer;ibli- to 
visual systems for high priority warnings. 

It can be Inferred from these data that pilots would like fi limited pri- 
ority assignment scheme, that all warnings which are current fdionld l>e displiyed 
somewhere, and that the pilot should decide the course ol .utlon r.ilher than 
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boing told whnt to do, llow«'Vor, within thono brond guldollnon, much morn work 
munt bo dono to d(*flmt the priority nonlgnmont nchrmion nnd to optlrolr.r* tho 
wnrnlnp, dlnplnyn, Cnndldnto nchoroort and dlapJayn munt bo thoroughly toatod In 
almulatlonn and In flight boforn they aro rocommondod for alrJlno uao. 

It haa alno boon ahown that proforrod cancollatlon optlona dopond on 
whothor tho warning la audltoryi vln»ml, or tactllo, aa wolJ an on tho priority 
or urgency of th<' warning. Tho data in figure 3 will aJlow a choice of the 
preferred cancel, latlon option, given the priority and warning mode. 

Finally, tho roaultn of thia atudy nhow that a ayatematlc, ubjoctlve mca- 
auremont of pilot prcforoncee for warning oyatem denign revealo conoiotency 
and otrong agrooroont among thin oiunple. of .line pllota. Wlille tho uoor cannot 
entirely dictate the aystem doalgn, oapeclally in airline cockplta where regu- 
latory and cout ccmaldoratlono arc so important, it would seem useful to 
Include input from exporionuod line pllota in the development of aircraft 
warning aystoma for civil transport aircraft. The aubjccts whose collective 
opinio' la represented by the date proaented here have much and varied experi- 
ence dying in different environments and aircraft types, and this should be 
given duo regard in the design of future warning syatema. 
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TAULK l,~ BACKGHOUNU Al'IU liJiPlilUENCli OF 50 HLOTO 


ppsmONS. FLOWN 

AIRLINFr. 


JJOUTFS.FE,OWN, 

CAPTAIN 13 

AMERICAN 

n 

INTERNATIONAL 

FIRST OFFICER 30 

PAN AM 

8 

OOMtSTIC 

FLIGHT ENQINELH 14 

UNITFO 

7 

SHORT HAUL 

OTHER 3 

WESTERN 

b 

CHARTER 


1 LYING IKiCRS 

4 

EREIGHT 


TWA 

2 

1 ERHY 


HUGHES AIR 

1 



UIRDAIR 

1 



OTHtR/^URLOUOH ?. 


AGC (yt ) 

YOUNQCST MEAN OLDEST 
27 41.3 60 


TOTAL TIME (hf) 
LEAST MEAN MOST 
500 9,300 30,000 


TABLE II.- URGENCY SCALE USED IN QUESTIONNAIRE 


LEVEL 

TYPE OF PROBLEM 

1 

IMMEDIATE ACTION REQUIRED BY CREW TO SAVE 
AIRCRAFT 

2 

IMMEDIATE ACTION REQUIRED BY CREW AS SOON 
AS AIRCRAFT IS STABLE 

3 

ACTION REQUIRED AS SOON AS TIME AVAILABLE 

4 

ACTION REQUIRED LATER IN THE FLIGHT' 
FLIGHT PLANNING MAY BE AFFECTED 

5 

ABNORMAL EVENTS SIGNALED FOR INFO ONLY; 

NO ACTION REQUIRED MAY AFFECT FLIGHT PLANNING 
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TAIILK UI.- WAKNlNfi Ml-ITIIODfi Af! )J1;F.II)1 :Ij IN QUJ.NTUtNNAi RK 


TYPE 

tXAMI'l I S 

MUSICAL SOUNDS^ 

TONES, CHIMES, CHORDS 

OTHER NON 
SPLEGH SOUNDS 

HI I.LR. CLACK! RS, HORNS, RIM^t HS 

VOICE 

ELECTRONIC SPEECH (LIKE HAL IN ?0()1 
OR LIKl OPW8 VOICE) 

LIGHTS WITH 
LABELS 

LIGHTS WITH PRINTED LABELS, ALL 
COLORS, HTCADY OR FLASHING 

UQHT8 WITH 
NO LABELS 

RAINTWLABtLEO LIGHfsTA^ COLORS; 
STEADY OR FLASHING 

TEXTl^TEiSAGEs'’ 

aIIphanumeric 

SCREEN OR AN ALPHANUMERIC DISPLAY 
BLOCK 

Tlaqs' 

"me^anIcal flags iN flight instrumeFtsT” 

DISPLAY COVERS, DOLL'S EYES 

TACTILE 

STICK SHAKERS, RUDDER SHAKERS, SEAf 
BOUNCERS 


TABLE IV.- TYPICAL PREFERENCE GRID FOR WARNING METHODS 
BO FALSE ALARMS PER 1 REAL ALARM 


URGENCY LEVEL 

MUSICAL 

SOUND 

1 

OTHER 

SOUND 

2 

VOICE 

3 

LABELED 

LIGHT 

4 

UNLABELED 

LIGHT 

5 

TEXT 

MESSAGE 

6 

FLAG 

7 

TACTILE 

8 

1 IMMEDIATE ACTION 
TO SAVE AIRCRAFT 









2 IMMEDIATE ACTION 
AFTER AIRCRAFT 
STABLE 









3 ACTION WHEN 
POSSIBLE 









4 ACTION LATER 









8 NO ACTION/ 
INFORMATION 
ONLY 
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TAIJLIi V.- PRKPKKKNCH GRID USED 


FOR FEASIIINO/NOT FEAfUlING 


AND AUDIO AEERT QUESTION 


WARNING SYSTEM 

1 

IMMEDIATE 

ACTION 

2 

ACTION 

WHEN 

STABLE 

3 

ACTION 

WHEN 

STABLE 

4 

ACTION 

LATER 

B 

INFO 

ONLY 

WARNING ON BOTTOM LINE OF 
CRT- ^NOT FLASHING 






WARNING ON BOTTOM LINE OF 
CRT- -FLASHING S/sec 




■ 1 


WARNING ON WHOLE CRT SCREEN- 
-NOT FLASHING. BUT WHATEVER WAS 
DISPLAYED BEFORE IS REMOVED 






WARNING ON WHOLE CRT SCREEN- 
-FLASHINQ. AND WHATEVER WAS 
DISPLAYED BEFORE IS REMOVED 






SINGLE lInE OF 12 3/4'' HIGH 
ALPHANUMERICS- -NOT FLASHING 






SINGLE LINE OF 12 3/4" HIGH 
ALPHA'.UMERICS- -FLASHING 3 
TIMES/sec 







TABLE VI.- SEMANTIC DIFFERENTIAL 
CONCEPTS USED 


Altitude alert tone 
3/4-ln. -square yellow light, flashing 
3/4-in, -high alphanumeric display, 
not flashing 

3/4-lu. -diameter red light, flashing 
3/4-ln, -square yellow light, not 
flashing 

Synthesized speech 
3/4-ln, -diameter red light, not 
flashing 

3/8-ln,-high lettering on a CRT 
Whoop, whoop, pull up, pull up 
ATC controller 

Mechanical flag in glides lope 
indicator 
Engine fire bell 

3/8-ln, blue light, not flashing 

Stick shaker 

SEL CAL tone 

VASl lights 

REIL lights 

Gear horn 


TABLE VII.- SEMANTIC DIFFERENTIAL 
SCALES USED 


Startling 


Tranqulllzlng 

Informative 

— 

Dls:ractlng 

Good 


Bad 

Ugly 


Beautiful 

Soft 


Hard 

Strong 


Weak 

Worthless 


Valuable 

Loud 

— 

Soft 

Unpleasant 

— 

Pleasant 

Hot 

— 

Cold 

Nice 


Awful 

Dark 

— 

Bright 

Active 


Passive 

Noisy 

— 

Quiet 

Safe 


Dangerous 

Alerting 

-- - 

Imperceptible 

Annoying 

— 

Soothing 


J 
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VOICE WARNING 

PRIORITY 1 - ACTION NOW 



1:50 1:1000 

FALSE-ALARM RATE 


Voice warning — priority 1 cell of warning 
method preferences figure. 
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VOICE WARNING 

PRIORITY 1 - ACTION NOW 
PRIORITY 2 - ACTION NOW 
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K CANCEL BUTTON 

□ AUTOMATIC AFTER 
5 seconds 

0 AUTOMATIC AFTER 
10 seconds 

□ NONCANCELLABLE 

E3 DON'T USE THIS 
WARNING METHOD 

□ NO PREFERENCE 





A5 A10 NC DU 
CANCELLATION OPTION 


Figure 4.- Voice warning — priority 1 or 2 cell of 
cancellation options figure. 

(%) VISUAI niSt'l AV 
ISPKIMARY 



Figure 5.- System logic preferences. 
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AFFIRMATIVE RE^ONSES 



1 2 3 4 D 

PRIORITY OP WARNING 

NOT FLASHING FLASHING 

"SINULE IINE OE TWELVE 3/4 " HIGH ALPHANUMERICS . , 

Figure 8.- Preferences for sliTglc* line of alphanumerics 



NO PRIOR VOICt WARNING AFTLM VOICE WARNING 
QUEST N 25 DEMON 25 

Figure 9.- Pilot acceptability of proposed 
voice warning nyoLoui. 


SYSTEM COMPONENT 


VISUAL STATUS 
DISPEAY 

VOICE CANCEL 
BUTTON 

REPLACE ALL AURAI- 
WITH VOICE 

VISUAL ON/OPF 
OPTION 

DIFFERENT WORDING 
EACH HAZARD 

SPEAKER AVC 


HEADPHONE AVC 



PERCENT OF “ESSENTIAL" JUDGMENTS 


Figure 10.- Percent of "essential" judgments for 
each component of pioposed voice warning 
system. 
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Figure 11.- Pilot prcforrncoM for repetition 
ot vo.io(‘ wnrningM, N ~ 50. 
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REPETITION OF WARNINGS SHOULD DEPEND ON 
URGENCY-PRIORITY 


PCRCCNTOF 

PILOTS 



Figure 12,- Percent ol "yes" and "no" reuponses for 
question regarding effect of warning urgency on 
preferred type of voice warning repetition. 
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Figure 13#"^ Pilot pret erencen lor ubob of electronic 
voice in cockpit. 
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3/H" \Ui\\\ U ITfRINnON ACM? 

3M" HIOH AMMlANliM! MIC U|SI’|-AV, N()| ( | AUMINH 


BTAMTUNn 

INrORMATIVf 

QOOP 

UQt-Y 

SOFT 

STRONG 

WORTHU.SS 

LOUD 

UNPLf.ASANT 
HOT 
NICE 
DARK 
ACTIVE 
NOISY 
SAFE 

ALERTING 
ANNOYING 
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DIfilHACIINri 
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HARD 
Wt AK 
VAIUAHII 

son 

PLtASANI 

COLD 

AWFUL 

BRIGHT 

PASSIVl 

QUIET 

DANGEROUS 

IMPERCEPTIBLE 

SOOTHING 


3/4'* DIAMETER RED LIGHT, FLASHING 

— — - - 3/4" SQUARE YELLOW LIGHT. FLASHING 



TRANQU1LI21NG 

DISTR^CTING 

BAD 

BEAUTIFUL 

HARD 

WEAK 

VALUABLE 

SOFT 

PLEASANT 

COLD 

AWFUL 

BRIGHT 

PASSIVE 

QUIET 

DANGEROUS 

IMPERCEPTIBLE 

SOOTHING 


Figure 14*^ Semantic differential average responBes, N 
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3/4" HIGH ALPHANUMERIC DISPLAY, NOT ELASHINO 
SYNTHESIZED SPEECH 
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Figure 14.- Continued 
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Figure 14 Concluded* 
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